There is a common smear labeled against secessionists that you must be pro-slavery to be a secessisonist. This is wildly untrue. Tom Woods covers Lysander Spooner who was a dedicated abolitionist and a proponent of secession.
Source: Ep. 322 Lysander Spooner: Anti-Slavery, Pro-Secession | Tom Woods