How not to Stitch Representations to Measure Similarity: Task Loss Matching versus Direct Matching

András Balogh, Márk Jelasity · 2024

Read How not to Stitch Representations to Measure Similarity: Task Loss Matching versus Direct Matching by András Balogh, Márk Jelasity (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11299.txt.

Description

András Balogh and Márk Jelasity argue that a popular way of stitching neural networks misleads similarity measurement. Model stitching joins two half networks through a learned mapping, and good stitched performance is read as functional similarity. The usual training method, task loss matching, optimizes the mapping on the task with both halves frozen.

Task loss matching can report very high similarity between distant layers with different known functions. It can even rate a layer as more similar to another layer than to itself. The cause is that the method creates out of distribution representations to boost task scores. Direct matching, which minimizes distance between stitched representations, avoids these failures. The comparison warns against a widely used index.

Catalog record

CREATORAndrás Balogh, Márk Jelasity
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11299

Browse the full library · About the Tower of Babel