Use of different set of tweets

[PickHub]

Hey,

I have a a few basic question: How do I tag my own set of tweets?

With what parser do I have to parse the data?

And how do I train the model on my own dataset?

Any help is much appreciated!

[Sentimentron]

Sure, my recommendation is to use twokenize.py to tokenize your tweets, possibly use the existing model to generate some initial labels and then manually correct them. You can quite easily retrain the existing model on your new dataset by changing the input_path parameter in lstm.py and then the final, uncommented line of [train.sh] (https://github.com/Sentimentron/Dracula/blob/2016-04-16/train.sh) contains the hyper-parameters I used for the final model. One of the other branches (currently called python-3, although it doesn't actually run on Python 3) also swaps out the initial layer for a convolutional one, which trains a bit faster and performs a bit better.

I'm happy to help if you need further assistance.

[PickHub]

Thanks a ton for the detailed answer.

I haven't found the method that actually adds the sentiment score to the tweets yet. Is that also implemented in lstm.py?

I already trained the model with Gate-Train.conll before. When I tried running server.py I got two syntax warnings repeatedly:

server.py:167: SyntaxWarning: name 'model' is assigned to before global declaration global model model options {'optimizer': <function adadelta at 0x7f749a8fcc08>, 'word_layers': encoder': 'lstm', 'letter_layers': 2, 'patience': 10, 'reload_model': 'lstm_model.00, 'dispFreq': 10, 'decay_c': 0.0001, 'use_dropout': True, 'char_dict': {}, 'wordpz', 'valid_batch_size': 64, 'pos_dict': {}, 'noise_std': 0.0} max length 140 server.py:55: UnicodeWarning: Unicode equal comparison failed to convert both argu being unequal word_dict[w.decode('utf8')] = word_dict[w] Continuing with model... Starting server... server.py:167: SyntaxWarning: name 'model' is assigned to before global declaration global model model options <...> max length 140 server.py:55: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal word_dict[w.decode('utf8')] = word_dict[w] Continuing with model... Starting server... WARNING:werkzeug: * Debugger is active!

When I run eval.py with Brown.conll I get a lot of terminal outputs stating "WRONG: 'string' JJ should be P/N..." and the accuracy is around 0.4. Is there something wrong with my trained model? Because I didn't change Brown.conll.

Also is it fairly easy to run the training on multiple gpus or ist that something that can be done easily?

[Sentimentron]

Ah, OK, so Brown.conll actually has a different tagset to Gate-Train.conll - there is a script somewhere that attempts to retag it so that it's more similar. There are a variety of sentiment branches too, unfortunately they do seem to be missing a process_training_data.py file in Data/. I'll check that file in as soon as I find it.