How not to Stitch Representations to Measure Similarity: Task Loss Matching versus Direct Matching
András Balogh, Márk Jelasity · 2024
Read How not to Stitch Representations to Measure Similarity: Task Loss Matching versus Direct Matching by András Balogh, Márk Jelasity (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11299.txt.
Description
András Balogh and Márk Jelasity argue that a popular way of stitching neural networks misleads similarity measurement. Model stitching joins two half networks through a learned mapping, and good stitched performance is read as functional similarity. The usual training method, task loss matching, optimizes the mapping on the task with both halves frozen.
Task loss matching can report very high similarity between distant layers with different known functions. It can even rate a layer as more similar to another layer than to itself. The cause is that the method creates out of distribution representations to boost task scores. Direct matching, which minimizes distance between stitched representations, avoids these failures. The comparison warns against a widely used index.
Catalog record
| CREATOR | András Balogh, Márk Jelasity |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11299