https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

http://purl.org/dc/terms/creator creator

https://w3id.org/np/RAoSadUw99... RAoSadUw99

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance

http://www.w3.org/ns/prov#wasAttributedTo wasAttributedTo

https://w3id.org/np/RAoSadUw99... RAoSadUw99

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance

http://www.w3.org/ns/prov#wasAttributedTo wasAttributedTo

https://orcid.org/0000-0002-0085-6496 0000-0002-0085-6496

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

http://www.w3.org/2000/01/rdf-schema#comment comment

Merging models trained for long with WIDEN When models were trained on a lot of data they diverged further from the baseline (e.g. in continual pretraining for additional languages), current merging methods underperform in this setting https://alphaxiv.org/pdf/2408.03092 @AlibabaGroup https://twitter.com/LChoshen/status/1823002789217493392/photo/1 How do you do that? Let's assume we update a matrix with a few models. Pick a pretrained model and consider the rest of the models as diff from it (task vectors) Normalize the row of each model, separating the normalization factor (magnitude) and direction (row) Now we weigh every row by how much it changed (higher = better) and average all together + some trick to sometimes keep the original weight so weights might not sum to 1. You can see how this follows recent findings about direction and size (e.g. https://x.com/prateeky2806/status/1727589818618523783) While the results in "just" merging are not changing that much, merging with a continually trained model (Sailor) that added many languages look quite good! https://twitter.com/LChoshen/status/1823002796259791276/photo/1 Criticism (@askalphaxiv didn't upload comment): There is a vast overclaiming calling Sailor a different pretrained model. Quite complex, hard to know if it will generalize and they only show a specific model.

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

https://schema.org/keywords keywords

large-language-models

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

https://schema.org/keywords keywords

Sailor

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

https://schema.org/keywords keywords

WIDEN

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

https://schema.org/keywords keywords

model-merging

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

https://schema.org/keywords keywords

weight-disentanglement

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance

http://www.w3.org/ns/prov#wasGeneratedBy wasGeneratedBy

https://w3id.org/np/RAPwHYQQtX...#activity activity

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

http://purl.org/spar/cito/discusses discusses

https://arxiv.org/abs/2311.13171 2311.13171

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

http://purl.org/spar/cito/discusses discusses

https://www.alphaxiv.org/pdf/2408.03092 2408.03092

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

http://purl.org/spar/cito/discusses discusses

https://x.com/prateeky2806/status/1727589818618523783 status/1727589818618523783

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its assertion http://www.nanopub.org/nschema#hasAssertion assertion

http://purl.org/spar/cito/includesQuotationFrom includesQuotationFrom

https://x.com/prateeky2806/status/1727589818618523783 status/1727589818618523783

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance

http://www.w3.org/ns/prov#wasAssociatedWith wasAssociatedWith

https://x.com/LChoshen LChoshen

Sensenets

2024-09-03T21:16:16.131Z

https://w3id.org/np/RAPwHYQQtX... RAPwHYQQtX

links a nanopublication to its provenance http://www.nanopub.org/nschema#hasProvenance provenance

http://www.w3.org/ns/prov#linksTo linksTo

https://x.com/LChoshen/status/1823002789217493392 status/1823002789217493392

Sensenets

2024-09-03T21:16:16.131Z