Attended Hugging Face Transformers Workshop.
Highlights:
Highlights:
- It started describing what attention mechanisms are and how is the architecture
- Showcased the concepts of Transformers, why and where are used. Also it was described where architectures are used or focused on the side of encoder or decoder. Such as BERT, T5 and GPT.
- Pre-training to train of a big corpus is usually done by big companies, and what you could do is perform Fine-Tuning in your own datasets.
- The main challenges:
- Language Barriers: such as being trained on English or European languages
- Data hungry: Pre train modeling required huge amount of data and it usually cost a lot of money
- Black boxes like Neural Nets, and we don't know exactly what is happening in terms of causality
- Biases: as the models are trained from the internet it comes with Bias, it can lean towards, stereotypes, racism etc.
- The workshop was based on demonstrating and explaining Hugging Face ecosystem where you can basically do/use tasks such as:
- Pipelines: a high level abstraction that given an input can give you predictions, it comes with a default model but you can set it to your own
- Datasets:
- You could have tasks performing the following:
- Entity Recognition: detects entities on a text. It can be useful on performing entity extraction and automation
- Question answering: having a text and a question, the model will provide an answer extracted from the input
- Summarization: given a text, it can generate a summary from it.
- Zero-shot classification: given a text and classes, the model is able to infer the probabilities related to the compatibility of the class.
It was also shown how you can work with :
- Datasets library, you could download a dataset from the hub and work with it in a matter of seconds using a pipeline.
- Spaces: a good way to write an application and have it in your own space, where you can basically write a blog or do a demo.
Model we created based on Amazon reviews corpus: https://huggingface.co/laurauzcategui/xlm-roberta-base-finetuned-marc-en