你可以使用position：

p <- ggplot(mtcars, aes(factor(vs), mpg, colour = factor(am))) +  
     geom_boxplot() +
     stat_summary(fun.data = give.n, geom = "text", fun.y = median,
                  position = position_dodge(width = 0.75))
p

position_dodge()的width参数控制水平轴上的位置。0.75是最佳点，请查看它如何适用于不同数量的分组：

p2 <- ggplot(mtcars, aes(factor(vs), mpg, colour = factor(cyl))) + 
      geom_boxplot() +
      stat_summary(fun.data = give.n, geom = "text", fun.y = median, 
                   position = position_dodge(width = 0.75))
p2

可以使用geom_text代替stat_summary，请参考以下问题：ggplot2 add text on top of boxplots。
这是一个如何使用观测数的示例：

# Create an aggregate of median & count
> cts <- merge(aggregate(mpg ~ cyl + am, mtcars, length), 
               aggregate(mpg ~ cyl + am, mtcars, median), 
               by=c("cyl", "am"))
# Rename the col names to fit with the original dataset..
> names(cts) <- c("cyl", "am", "count", "mpg")
# As alexwhan suggested, position_dodge helps with positioning
# along the x-axis..
> ggplot(mtcars, aes(factor(cyl), mpg, colour = factor(am))) + 
  geom_boxplot(position = position_dodge(width=1.0)) + 
  geom_text(data = cts, aes(label=count), 
            position=position_dodge(width=1.0))