As the initial box of the pedestrian detection algorithm, anchor can solve the problem of pedestrian translation and alleviate the problem of pedestrian scale variation. At present, the pedestrian detection algorithms are usually based on the anchor. However, the usage of the anchor requires careful adjustment of the hyper-parameters of the anchor, such as the scale and aspect ratio of the anchor, which have a great impact on the detection performance. To circumvent this problem, we present a pedestrian detection algorithm based on an anchor-free loss function. By fusing the features of all detection branches of feature pyramid network(FPN), the algorithm does not need to set an effective training scale range for each detection branch of FPN in the training process. In addition, a SA(scale attention) module is proposed to fuse all the detection branch features of FPN, so that appropriate weights can be adaptively assigned to the region of interest(ROI) features of different scales corresponding to pedestrians when the network detects a certain scale of pedestrian. Experiment results show that the proposed pedestrian detection algorithm not only realizes anchor-free, thus circumvent the super-parameter adjustment problem of the anchor, but also outperforms other pedestrian detection algorithms, achieves 9.19% MR² which is the best of state-of-the-art results on CityPersons dataset.