hok@lemmy.dbzer0.comOPtoFree Open-Source Artificial Intelligence@lemmy.world•Is there anything that makes training a translation task easy?English
0·
1 year agoThanks, the quickstart guide was straightforward to follow. Do you have any suggestions on how to do word splitting with code, if any? For example, on a test run, I found that the model was not able to synthesize unique constants correctly even though this test run consisted only of obvious “a to b” relationships.
Thanks for the tips. After doing a bunch of searching, I found that what I needed was BPE, or byte-pair encoding. This allows the token set to contain sub-word sequences, which lets the tokenizer represent a unique constant like
0x0373
as['__sow', '0x', '03', '73', '__eow']
.