If something such as cloth or rope frays, or if something frays it, its threads or fibres start to come apart from each other and spoil its appearance.
If you say that something is fraying at the edges or is fraying around the edges, you mean that it has an uncertain or unsteady quality, for example because it is gradually being spoiled or destroyed.