Summer Pockets is set on an isolated, rural, and peaceful island on the Seto Inland Sea called Torishirojima, which has a population of about 2,000 people. The protagonist, Hairi Takahara, grew up in an urban setting, but after an unpleasant incident, he uses the recent death of his grandmother as an excuse to come to the island to take care of her estate sale. Once there, he gets to know four girls, who are the focus of the story: Shiroha Naruse, who forgot her summer vacation; Ao Sorakado, who is pursuing the legends of the island; Kamome Kushima, a high-class girl looking for a pirate ship; and Tsumugi Wenders, a younger girl trying to find herself.