Autoencoders for Constructing Language

Datasets

Datasets are saved as tensor datasets where each sample is represented by a single dimensional tensor of tokens. Samples should be padded to the same length by using padding tokens (PAD_ID). Datasets should be saved as PyTorch .pt files.

Steps to obtain a usable dataset from a text file corpus:

1. Prepare a file (i.e. corpus.txt) where each different line represents a different sentence. Keeping punctuation and not lemmatizing is ok!
2. Train a BPE tokenizer on the corpus by using the command: python utils/tokenizer.py corpus.txt --save_path path/to/output

• Some optional parameters to this procdedure include:
  • --max_len L: filter all sentences with more tokens than L after adding end and start of sentence tokens
  • --vocab_size V: train tokenizer to output no more than V tokens
  • [--val_split, --test_split] x: split off x (between 0 and 1) of the corpus to be in the validation/test sets

The tokenizer.py command saves multiple files to the save_path:

• [train, val, test].pt are split PyTorch Datasets that store one tokenized sentence per entry, all padded to the same length.
• tokenizer.model stores the weights of the trained tokenizer
• tokenizer.vocab stores the indices and mapped characters for each token

Training

A training job can be started by creating a .yaml config file and running train.py spec.yaml.

An example .yaml file is shown below:

lang:
  input:
    vocab_size: 512
    seq_len: 64
  output:
    vocab_size: 64
    seq_len: 32
dataset:
  train: ./starwars/datasets/complete/train.pt
  val: ./starwars/datasets/complete/val.pt
model:
  path: ./starwars/models/test.pt
  d_model: 256
  nhead: 4
  num_encoder_layers: 3
  num_decoder_layers: 3
  dim_feedforward: 2048
  temperature: 0.1
train:
  lr: 0.0001
  weight_decay: 0.0001
  dropout: 0.1
  discrete: False
  batch_size: 256
  epochs: 10
  clip_grad_norm: 1.0
predict:
  tokenizer: ./starwars/datasets/complete/tokenizer.model

Model specification parameters include:

• lang.input denotes the language of the training data
• lang.output denotes the constructed language generated by the model
• vocab_size denotes the range of tokens pertaining to the language
• seq_len denotes the maximum length of a sentence unit in the language
• temperature specifies how "random" we want samples to be (higher means more random)
• train.path is the path where the finished model will be saved
• discrete specifies whether or not samples from the constructed language are discrete tokens or Gumbel-softmax combinations
• tokenizer is the path to the tokenizer trained during preprocessing

TODO Add better training tools:

• Saving the top 1 model by validation loss
• Generating plots of loss vs batch
• Creating tools for hyperparameter tuning

Evaluation

TODO