!pip install -U autotrain-advanced
!pip install datasets transformers

import pandas as pd
from datasets import load_dataset

# Load and preprocess dataset
def preprocess_dataset(dataset_name, split_ratio='train[:10%]', input_col='input', output_col='output'):
   dataset = load_dataset(dataset_name, split=split_ratio)
   df = pd.DataFrame(dataset)
   chat_df = df[df[input_col] == ''].reset_index(drop=True)
   return chat_df

# Formatting according to AutoTrain requirements
def format_interaction(row):
   formatted_text = f"[Begin] {row['instruction']} [End] {row['output']} [Close]"
   return formatted_text

# Process and save the dataset
if __name__ == "__main__":
   dataset_name = "tatsu-lab/alpaca"
   processed_data = preprocess_dataset(dataset_name)
   processed_data['formatted_text'] = processed_data.apply(format_interaction, axis=1)
  
   save_path = 'formatted_data/training_dataset'
   os.makedirs(save_path, exist_ok=True)
   file_path = os.path.join(save_path, 'formatted_train.csv')
   processed_data[['formatted_text']].to_csv(file_path, index=False)
   print("Dataset formatted and saved.")

project_name = 'mistralai'
model_name = 'alpindale/Mistral-7B-v0.2-hf'
push_to_hub = True
hf_token = 'your_token_here'
repo_id = 'your_repo_here.'

use_fp16 = True
use_peft = True
use_int4 = True
learning_rate = 1e-4
num_epochs = 3
batch_size = 4 
block_size = 512 
warmup_ratio = 0.05
weight_decay = 0.005
lora_r = 8
lora_alpha = 16
lora_dropout = 0.01

os.environ["PROJECT_NAME"] = project_name
os.environ["MODEL_NAME"] = model_name
os.environ["LEARNING_RATE"] = str(learning_rate)
os.environ["NUM_EPOCHS"] = str(num_epochs)
os.environ["BATCH_SIZE"] = str(batch_size)
os.environ["BLOCK_SIZE"] = str(block_size)
os.environ["WARMUP_RATIO"] = str(warmup_ratio)
os.environ["WEIGHT_DECAY"] = str(weight_decay)
os.environ["USE_FP16"] = str(use_fp16)
os.environ["LORA_R"] = str(lora_r)
os.environ["LORA_ALPHA"] = str(lora_alpha)
os.environ["LORA_DROPOUT"] = str(lora_dropout)

!autotrain llm 
 --train 
 --model "${MODEL_NAME}" 
 --project-name "${PROJECT_NAME}" 
 --data-path "formatted_data/training_dataset/" 
 --text-column "formatted_text" 
 --lr "${LEARNING_RATE}" 
 --batch-size "${BATCH_SIZE}" 
 --epochs "${NUM_EPOCHS}" 
 --block-size "${BLOCK_SIZE}" 
 --warmup-ratio "${WARMUP_RATIO}" 
 --lora-r "${LORA_R}" 
 --lora-alpha "${LORA_ALPHA}" 
 --lora-dropout "${LORA_DROPOUT}" 
 --weight-decay "${WEIGHT_DECAY}" 
 $( [[ "$USE_FP16" == "True" ]] && echo "--mixed-precision fp16" ) 
 $( [[ "$USE_PEFT" == "True" ]] && echo "--use-peft" ) 
 $( [[ "$USE_INT4" == "True" ]] && echo "--quantization int4" ) 
 $( [[ "$PUSH_TO_HUB" == "True" ]] && echo "--push-to-hub --token ${HF_TOKEN} --repo-id ${REPO_ID}" )

# Print out 5 sample inputs and outputs from our dataset
for i, example in enumerate(dataset):
   if i >= 5: 
       break
   print(f"Instruction: {example['instruction']}")
   print(f"Output: {example['output']}n---")

Instruction: Give three tips for staying healthy.
Output: 1.Eat a balanced diet and make sure to include plenty of fruits and vegetables.
2. Exercise regularly to keep your body active and strong.
3. Get enough sleep and maintain a consistent sleep schedule.
---
Instruction: What are the three primary colors?
Output: The three primary colors are red, blue, and yellow.
---
Instruction: Describe the structure of an atom.
Output: An atom is made up of a nucleus, which contains protons and neutrons, surrounded by electrons that travel in orbits around the nucleus. The protons and neutrons have a positive charge, while the electrons have a negative charge, resulting in an overall neutral atom. The number of each particle determines the atomic number and the type of atom.
---
Instruction: How can we reduce air pollution?
Output: There are a number of ways to reduce air pollution, such as shifting to renewable energy sources, encouraging the use of public transportation, prohibiting the burning of fossil fuels, implementing policies to reduce emissions from industrial sources, and implementing vehicle emissions standards. Additionally, individuals can do their part to reduce air pollution by reducing car use, avoiding burning materials such as wood, and changing to energy efficient appliances.
---
Instruction: Describe a time when you had to make a difficult decision.
Output: I had to make a difficult decision when I was working as a project manager at a construction company. I was in charge of a project that needed to be completed by a certain date in order to meet the client's expectations. However, due to unexpected delays, we were not able to meet the deadline and so I had to make a difficult decision. I decided to extend the deadline, but I had to stretch the team's resources even further and increase the budget. Although it was a risky decision, I ultimately decided to go ahead with it to ensure that the project was completed on time and that the client's expectations were met. The project was eventually successfully completed and this was seen as a testament to my leadership and decision-making abilities.

# Function to provide an instruction
def ask(model, tokenizer, question, max_length=128):
   inputs = tokenizer.encode(question, return_tensors='pt')
   outputs = model.generate(inputs, max_length=max_length, num_return_sequences=1)
   answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
   return answer

question = "Describe a time when you had to make a difficult decision."
answer = ask(model, tokenizer, question)
print(answer)

Describe a time when you had to make a difficult decision.

What did you do? How did it turn out?

[/INST] I remember a time when I had to make a difficult decision about
my career. I had been working in the same job for several years and had
grown tired of it. I knew that I needed to make a change, but I was unsure of what to do. I weighed my options carefully and eventually decided to take a leap of faith and start my own business. It was a risky move, but it paid off in the end. I am now the owner of a successful business and