问题是什么？

有一天我注意到对于非常长的提示，结果非常差，所以我用不同的提示测试了几次，发现如果 total_tokens ( prompt_tokens + completion_tokens ) 超过2048,它会输出非常差的结果，如下所示：

$ curl -X POST http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
"model": "llama3:8b",
"stream": false,
"max_tokens": 300,
"messages": [
{
"role": "user",
"content": "Spot saw the shiny car and said, Wow, Kitty, your car is so bright and clean! Kitty smiled and replied, Thank you, Spot. I polish it every day. After playing with the car, Kitty and Spot felt thirsty. They found a small pond with clear water. They drank the water and felt very happy. They played together all day and became best friends. Once upon a time, in a big forest, there lived a rhinoceros named Roxy. Roxy loved to climb. She climbed trees, rocks, and hills. One day, Roxy found an icy hill. She had never seen anything like it before. It was shiny and cold, and she wanted to climb it. Roxy tried to climb the icy hill, but it was very slippery. She tried again and again, but she kept falling down. Roxy was sad. She wanted to climb the icy hill so much. Then, she saw a little bird named Billy. Billy saw that Roxy was sad and asked, Why are you sad, Roxy? Roxy told Billy about the icy hill and how she couldnt climb it. Billy said, I have an idea! Lets find some big leaves to put under your feet. They will help you climb the icy hill. Roxy and Billy looked for big leaves and found some. Roxy put the leaves under her feet and tried to climb the icy hill again. This time, Roxy didnt slip. She climbed and climbed until she reached the top of the icy hill. Roxy was so happy! She and Billy played on the icy hill all day. From that day on, Roxy and Billy were the best of friends, and they climbed and played together all the time. And Roxy learned that with a little help from a friend, she could climb anything. Once upon a time, in a small yard, there was a small daisy. The daisy had a name. Her name was Daisy. Daisy was very small, but she was also very happy. One day, Daisy saw a dog. The dog was big and had a name too. His name was Max. Max liked to play in the yard. Daisy liked to watch Max play. Max and Daisy became friends. Every day, Max would come to the yard to play. Daisy would watch and smile. They were very happy together. And even though Daisy was small, she knew that she had a big friend in Max. Once upon a time, there was a thoughtful girl named Sue. Sue loved to help her mom around the house. One day, her mom asked her to wipe the table after they ate their lunch. Sue was happy to help. As Sue was wiping the table, she saw a pretty candle on the window sill. The candle was her moms favorite. Sue wanted to do something nice for her mom, so she said, Mom, can I light the candle for you? Her mom said, Yes, but be very careful. Sue carefully lit the candle and put it on the table. Her mom was so happy to see the pretty candle. They both sat and watched the candle burn. Sues mom said, Thank you, Sue, for being so thoughtful and careful. Sue felt proud that she could help her mom. The moral of the story is to always be thoughtful and careful when helping others. Once upon a time, there was a kind farmer. He had a big cow. The cow was sad. The farmer did not know why. One day, a little boy came to the farm. He saw the sad cow. The boy kneeled down to talk to the cow. Why are you sad, cow? he asked. The cow said, I am lonely. I want a friend. The kind farmer heard the cow. He wanted to help. So, he got another cow to be friends with the sad cow. The sad cow was happy now. They played together every day. And the kind farmer, the little boy, and the two cows all lived happily ever after. Once upon a time, there was a little girl named Lucy. She had a pet cat named Tom. They loved to play together in the big green park near their house. One sunny day, they went to the park to play. While playing, Tom saw a big sour lemon on the ground. He wanted to play with it, but when he touched it, it started to roll away. Tom ran after the lemon, trying to catch it. But as he ran, Tom got lost in the park. Lucy looked around, but she could not find Tom. She was very sad. Lucy did not give up. She searched the park for her friend. At last, she found him near a big tree. Tom was trying to catch the lemon, but it vanished into a hole in the ground. Tom was happy to see Lucy again. They hugged and went back home together. They had a fun escape in the park, but they decided to leave the sour lemon behind. Once upon a time, there was a little brown dog named Spot. He loved to play with his ball in the park. One sunny day, Spot saw a big goal on the other side of the park. He wanted to get his ball into the goal. Spot ran fast with the ball in his mouth. He tried to kick the ball into the goal, but he was too small. Spot started to struggle. He tried again and again, but the ball would not go in. Then, Spot had an idea. He asked his friend, a big brown horse named Buddy, for help. Buddy kicked the ball with his strong legs. The ball flew into the goal! Spot was so happy. He and Buddy played together all day long. Once upon a time, there was a little boy named Tom. He loved to play with his red ball. One sunny day, Tom went outside to play with his ball in the land near his home. Tom kicked the ball high in the sky. The ball went far, far away. Tom was sad because he could not find his ball. He walked and walked, looking for it. The land was big and sometimes dangerous. Tom knew he had to be careful. At last, Tom found his ball near a big tree. He was very happy. Tom knew he should not kick the ball too hard next time. He went back home, holding his ball tightly. Tom played safely in his yard, away from the dangerous land. Once upon a time, there was a big dog named Max. Max had a red collar that he wore every day. He loved to play and run in the park with his friends. One day, Max saw a cat on a tree. He wanted to be friends with the cat. So, Max tried to stretch up to reach the cat. But he was not tall enough. He tried again and again, but he just couldnt reach the cat. Max felt sad, but then he had an idea. He found a big box and put it under the tree. Max climbed on the box and stretched one more time. This time, he reached the cat! The cat and Max became good friends, and they played together every day. Once upon a time, there was a girl named Mia. Mia loved her jewelry. She had a big box full of pretty things. She liked to wear them all day. But at night, she had to sleep. One day, Mia met a talking cat named Tom. Tom was a tough cat, but he was nice. Tom said, Hi, Mia! I like your jewelry. Can I wear some too? Mia said, Yes, Tom. You can wear my jewelry, but we have to give it back before we sleep. So, Mia and Tom played together. They wore the jewelry and had fun. They pretended to be kings and queens. They laughed and danced. But soon, the sun went down, and it was time for bed. Mia said, Tom, we must give back the jewelry now. Its time to sleep. Tom gave back the jewelry and said, Thank you, Mia. I had fun today. They put the jewelry back in the box and went to sleep. Mia and Tom were happy, and they had sweet dreams. Once upon a time, there was a little boy named Tim. Tim had a big, orange ball. He loved his ball very much. One day, Tim met a girl named Sue. Sue had a pretty doll. Tim liked Sues doll, and Sue liked Tims orange ball. Tim and Sue thought about a trade. They would trade the ball for the doll. Tim was not sure. He loved his orange ball. Sue said, I promise to take care of your ball. You can play with it when you want. Tim said, I promise to take care of your doll too. They did the trade. Tim played with the doll, and Sue played with the orange ball. But then, Sues big dog came. The dog saw the orange ball and thought it was a toy. The dog bit the ball and it popped. Tim was very sad, and Sue was sorry. The trade was not good, and the orange ball was gone forever. Once upon a time, there was a little boy named Tom. Tom had a special belt that he loved to wear. One day, he could not find his belt and felt very sad. Toms mom saw him and asked, Why are you sad, Tom? Tom replied, I cant find my belt, and I miss it. His mom wanted to comfort him, so she said, Lets look for it together. They looked and looked, and finally, they found the belt under Toms bed. Tom was so happy, and his mom gave him a big hug. From that day on, Tom always remembered to keep his belt in a safe place. Once upon a time, in a big park, there was a fun slide. Kids loved to play on the slide all day. One day, a small boy named Tim came to the park to play. Tim was a generous boy who loved to share his toys with his friends. As Tim played on the slide, he saw a wreck near the park. The wreck was made of wood and old things. Tim thought it would be fun to make something new from the wreck. So, he asked his friends to help him. Together, Tim and his friends moved the wreck. They made a new slide from the wood. Now, the park had two slides for kids to play on. Tim and his friends were so happy. They played on the new slide all day long. Once upon a time, in a small house, there lived a kind and honest girl named Lily. She loved to bake cakes for her friends and family. One day, she made a big, yummy cake for her best friends birthday. Lily carefully put the cake in the oven to bake. She waited and waited until it was ready. When the cake was done, she had to remove it from the oven. She used big, thick gloves to take the hot cake out. Then, she let the cake cool down. After the cake cooled, Lily decorated it with sweet frosting and colorful sprinkles. She was excited to give the cake to her best friend. At the birthday party, everyone loved the cake and thanked Lily for her hard work. Lily felt happy and proud to be an honest friend who shared her yummy cake. Once upon a time, there was a pirate named Tom. Tom was an honest pirate who always told the truth. One day, Tom found a treasure map. He decided to look for the treasure with his friend, Sam. Tom and Sam sailed on their boat. They followed the map and found an island. On the island, they saw a big, scary pirate named Max. Max wanted the treasure too. Tom said, Max, we found the treasure first. Please let us have it. Max did not want to give up. But Tom and Sam were brave. They told Max, If you surrender and stop fighting, we can share the treasure with you. Max thought about it and agreed. They all worked together to find the treasure. In the end, Tom, Sam, and Max found the treasure and shared it. They learned that being honest and working together is better than fighting. And they all became good friends. One day, a young boy named Tim found a dull, round rock. He picked it up and looked at it. He thought it was not very fun, but he took it with him to the park. At the park, Tim saw a girl named Sue. She had a big, red ball. Want to play? Sue asked. Tim said, Yes, but all I have is this dull rock. Sue said, Lets try to roll it. They tried to roll the rock, but it was not easy. They pushed it and pushed it, but it would not go far. They were sad, but they did not give up. Then, they had an idea. They put the rock on a hill and let it go. The rock rolled down the hill very fast. Tim and Sue laughed and clapped their hands. They found out that even a dull rock can be fun when you roll it down a hill. Once upon a time, there was a graceful cat named Kitty. She loved to play with her ball and jump around the house. Kitty was very happy and liked to make her friends laugh. One day, Kitty saw a tap in the garden. She was very curious about it. She wanted to see if she could make water come out. So, she jumped onto the tap and tried to squeeze it with her paws. But nothing happened. Kitty tried again and again to squeeze the tap, but still no water came out. Then, her friend, a wise old dog named Rover, came to help her. Rover showed her how to turn the tap on, and water came splashing out! Kitty was so happy, and they both played in the water all day long. Once upon a time, there was a little boy named Tim. Tim loved candy more than anything else. One day, Tim saw a big candy store. He was very happy and ran to the store. Inside the store, Tim met a strange man. The man said, I have magic candy, but you must not eat it all. If you do, you will freeze! Tim took the candy and said, I will not eat it all. Tim went home and ate some of the candy. It was the best candy he ever had. But Tim did not listen to the man. He ate all the candy. Suddenly, Tim felt very cold. Tim started to freeze. He could not move or talk. His mom found him and cried, Oh no, my boy is frozen! And so, Tim stayed frozen forever, all because he did not listen to the strange man in the candy store. Once upon a time, there was a big, red barn. Inside the barn, there was a lot of hay. The hay was soft and yellow. It was a nice place for the animals to sleep. In the barn, there was a tall horse named Sam. Sam was a dependable friend to all the animals. He would stand by them and help them when they needed it. Everyone liked Sam because he was kind and strong. One day, a little chick got lost in the hay. The chick was scared and called for help. Sam heard the chick and went to find her. He stood by the hay and used his long legs to move it. Finally, he found the little chick and took her back to her mom. All the animals cheered for Sam, the dependable horse who stood by his friends. Once upon a time, there was a big, heavy alligator. He lived near a small pond. He was very hungry and wanted to eat something. One day, a little bunny came close to the pond. The alligator saw the bunny and thought, I can eat the bunny! The bunny saw the alligator and was scared. The alligator wanted to catch the bunny, but the bunny was too fast. The bunny ran away from the alligator as fast as he could. The alligator tried to catch the bunny, but he was too heavy and slow. The bunny got away and the alligator was still hungry. The alligator learned that he needed to be faster to catch the bunny. One day, a little boy named Tim went to play with his friend, Sam. They wanted to play a game with a ball. The game was to see who could get the best score. Tim threw the ball and Sam tried to catch it. Sam missed the ball and it went far away. Sam was angry. He said, I dont like this game. I want to lie down and rest. Tim said, Okay, lets lie down and rest. Then we can try again to get a good score. They lay down under a big tree and looked at the sky. They saw birds flying and clouds moving. After a while, they got up and played the game again. This time, Sam caught the ball and got a good score. Sam was happy and not angry anymore. Good job, Sam! said Tim. They played the game all day and had lots of fun. They learned that sometimes its good to take a break and try again. Once upon a time, there was a big whale. The whale lived in the deep blue sea. He had many friends, like fish and crabs. But he had a problem. He was very smelly. His friends did not like the smell. One day, the whale thought, I need to find a way to not be smelly. So, he swam and swam until he found a big rock. The rock was covered in green plants. The whale rubbed against the rock to clean himself. The green plants took away the smell. The whale swam back to his friends. They were all happy that he was not smelly anymore. They played together in the sea. The whale learned that it is important to be clean. And all the sea friends lived happily ever after. Once upon a time, there was a lively little boy named Tim. He loved to play and run all day. One day, Tim found a big bag of oats. He believed that if he ate the oats, he would be very strong. Tim ate a lot of oats every day. He felt stronger and stronger. His friends saw him and wanted to eat oats too. They believed that they would be strong like Tim. But one day, Tim ate too many oats. His tummy hurt a lot. He was not lively anymore. Tims friends felt sad for him. They learned that eating too many oats was not good. From that day on, they did not believe in eating lots of oats to be strong."
}
]
}'
---
...
  "usage": {
    "prompt_tokens": 1809,
    "completion_tokens": 292,
    "total_tokens": 2101
  }
...

好的，既然 total_tokens 是2101并且超过了2048,接下来会发生什么呢？

$ curl -X POST http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
"model": "llama3:8b",
"stream": false,
"max_tokens": 200,
"messages": [
{
"role": "user",
"content": "Hello?"
}
]
}'
---
#1
        "content": "Hello! You've never seen that is so cool! How can I'm happy eyes.\nIs there were busy building blocks, his mom and smiled\nHi-..."
#2
        "content": "Hi! Someone's car."
#3
        "content": "Hello! I'm so and your carattractive!"

你有没有注意到一些奇怪的事情？
正常的结果应该是这样的：

$ curl -X POST http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
"model": "llama3:8b",
"stream": false,
"max_tokens": 200,
"messages": [
{
"role": "user",
"content": "Hello?"
}
]
}'
---
#1
        "content": "Hello! It's nice to meet you. I'm here to help answer any questions or just chat if you'd like. What's on your mind today?"
#2 
        "content": "Hello! It's nice to meet you. I'm an AI assistant, and I'm here to help answer any questions or just chat with you. What brings you here today?"
#3
        "content": "Hello! It's nice to meet you. Is there something I can help you with, or would you like to chat?"

这是在超过2048个令牌之前正常的输出，这意味着模型在之前就已经坏了。它从未恢复到正常状态，并产生了非常不寻常的结果。这似乎是llama.cpp或ollama_llama_server中的一个非常严重的BUG,应该尽快修复。

操作系统

Linux

GPU

Nvidia

CPU

Intel

Ollama版本

0.1.43